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Executive Summary 


The project 


Research into Practice — Evidence-informed Continuing Professional Development in Rochdale was a 
pilot intervention aimed at supporting teachers to use evidence-based teaching and learning strategies 
to improve pupil progress. The project ran for one year (2014/2015) in ten primary schools in the 
Rochdale area, all of which are members of the Inspirational Professional Learning Community 
Network (IPLCN), and was delivered by a senior Continuing Professional Development (CPD) 
consultant based at one of the schools. It involved CPD sessions and direct consultant support 
designed to help teachers to: 


e have more positive views about the use of research for improving teaching and learning; 
e apply educational research findings in the classroom and at a strategic development level; and 
e establish a stronger culture of evidence-based enquiry and practice. 


In total, about 280 pupils were taught by participating teachers. The project aimed to improve pupil 
attainment as a longer-term outcome by improving pupils’ attitudes to learning, and by using evidence- 
based teaching and learning strategies such as metacognition, self-regulation and feedback. 


The principle objective of this study was to explore whether, and to what extent, research 
communication and engagement strategies had the potential to improve teachers’ use of, and 
attitudes towards, academic research to support pupils’ progress. The project was funded through the 
Education Endowment Foundation (EEF) Research Use in Schools grants round. It was jointly funded 
by the EEF, the Department for Education, and the Mayor’s London Schools Excellence Fund. 


| Key conclusions 


|: ug eee | 
1. There were some positive changes in teachers’ attitudes towards research during the course 


of the pilot. 


2. There was no evidence that teachers were more likely to use research to inform their teaching 
practice after being involved in the pilot. 


3. The project was very well received by teachers suggesting that this model may be a promising 
way of engaging teachers in evidence-based practice. 


4. Finding time for working collaboratively on implementing research evidence in practice was 
considered a challenge, but overall the requirements of the programme were feasible. 


5. Before a trial is considered, further thought should be given as to which elements of the project 
are essential for its efficacy, and whether a trial should test the project structure as a model for 
research dissemination or both the structure and content of the project as piloted. 


What are the findings? 


There were some positive changes in teachers’ attitudes to research during the course of the pilot as 
measured using a survey developed specifically for the EEF’s Research Use grants. These included 
an increase in the proportion of teachers positively disposed to academic research informing teaching 
practice, and a decline in teachers’ perceptions that academic research is not useful to teaching. 
Although these changes cannot be attributed confidently to the intervention without a comparison 
group, there was some indication that teachers with direct involvement in the programme may have 
experienced greater positive changes. This providess some evidence that the changes observed were 
related to the intervention. 
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Findings from qualitative interviews suggested that the intervention’s structure provided scope to 
embed the use of research evidence in practice because it overcomes barriers related to time and 
practical implementation. They also suggest that this structure will need to remain in place after the 
pilot ends if teachers are to continue to engage with research evidence in the long term. 


The programme largely ran as intended and was perceived very positively by participating staff. There 
was substantial buy-in from schools’ senior leadership teams facilitated by the project team’s effective 
engagement with senior leaders, helped by the fact that the schools were members of a pre-existing 
network of schools. For the programme to be successful without such an existing network, careful 
thinking will be needed about how to replicate the level of school engagement achieved in the pilot. 
Finding time for working collaboratively on implementing research evidence was also perceived to be 
a challenge in the pilot, but overall the requirements of the programme were feasible. 


We believe that the intervention is not yet ready to be evaluated in a trial. Further thought should be 
given to which elements of the project are essential for its efficacy and whether a trial should test the 
project structure as a model for research dissemination or both the structure and content of the project 
as piloted. Feasibility in schools without a pre-existing network would also need to be considered, 
alongside further clarification of the treatment group, outcome measures, and trial length. 


How was the evaluation of the pilot conducted? 


The pilot study included elements of formative evaluation, process evaluation, and quantitative data 
collection and analysis. Findings were shared with the delivery team as they became available in order 
to enable collaborative working and facilitate ongoing development of the intervention. 


A model describing how the intervention would work in practice was drafted by NatCen researchers in 
consultation with the CPD consultant following a workshop to identify the resources, activities, outputs, 
and intended outcomes of the programme. This was shared with participating teachers during training 
events. Teachers at participating schools were surveyed at the beginning and end of the academic 
year, and quantitative data from this survey was analysed to identify any changes. The process 
evaluation was based on depth interviews, observations of training events, case studies, and survey 
data. Interviews and observations took place throughout the academic year in order to capture 
experiences of participants as the intervention was being implemented. All schools took part in some 
process evaluation activities. 


The cost of the intervention was estimated at £74,759 per year, or at £267 per pupil. This is based on 
280 pupils being affected by the intervention activities across the ten participating schools. We 
estimate that as the number of pupils benefiting from the intervention increases in the programme’s 
subsequent years, the cost per pupil in its third year would reduce to an estimated £172. 


Question tale are) | Comment 
Is there evidence of ‘Vise There were improvements in teachers’ attitudes 
promise? towards research between baseline and follow-up. 


Was the approach The programme ran as intended and was perceived 


Yes 


feasible? positively by participating teachers. 
Is the approach ready to No The programme needs to be developed further 
be evaluated in a trial? before being evaluated in a full trial. 
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Introduction 


The aim of this pilot study was to evaluate the Research into Practice — Evidence-informed CPD in 
Rochdale programme (‘the programme’ or ‘the intervention’). The programme was delivered in ten 
primary schools in the Rochdale area, located in the North of England. It was funded by the Education 
Endowment Foundation as part of the Research Use in Schools round of projects" and was delivered 
during the 2014/2015 academic year. 


Intervention 


The programme was an intervention in ten primary schools in the Rochdale area, all of which were 
part of the Inspirational Professional Learning Community Network (IPLCN). The intervention aimed to 
develop teaching expertise and practice and ultimately improve educational outcomes by: 


e implementing a model of high quality CPD within participating schools by using research 
evidence to embed a collaborative learning culture and develop professional practice; and by 


e using research evidence on effective teaching and learning strategies—such as 
metacognition, learning cultures, self-regulated learning and feedback—to equip teachers with 
effective learning strategies, and pupils with the tools they need to achieve and progress. 


To achieve these aims, the CPD lead, with support from an external expert advisor, was responsible 
for developing and running the following components: 

e developing and leading CPD sessions; 

e undertaking termly school visits to each participating school; and 

e providing email and phone advice and guidance on request. 
The CPD lead was based at one of the participating schools and was funded by the project for 40 
days across the pilot year. The CPD lead was a qualified teacher and had been a professional 


development consultant for ten years previously, working across the schools in the Rochdale IPLCN, 
so she was already well known to the schools involved. 


2014/2015 Intervention activities 


Autumn term e CPD sessions, 2 full-day and 1 half-day 
e 2school visits from CPD lead 


e On-going classroom implementation 


Spring term e CPD sessions: 1 full-day and 1 half-day 
e Third school visit from CPD lead 


e On-going classroom implementation 


Summer term e 2 half-day CPD sessions 
e School peer visits to share practice 


e On-going classroom implementation 


There were six strands to the intervention: 


‘ https://educationendowmentfoundation.org.uk/news/eef-launches-15-million-fund-to-improve-use-of-research- 


in-schools/ 
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e CPD sessions 


A series of CPD events were delivered targeted at the Maths Subject Lead, the Literacy 
Subject Lead, and a classroom teacher from each participating primary school. The 
attendance of all three staff from each school was considered essential to the success of the 
model because of the importance placed on collaborative CPD, and on embedding a learning 
culture within the school. 


In total, three full days and four half-day CPD sessions were delivered over the course of the 
year. The purpose of these was to provide participants with an introduction to research 
evidence on metacognition, self-regulated learning, and feedback (chosen because of existing 
research evidence on their classroom efficacy’), and to support teachers to work 
collaboratively to plan and implement these approaches in the classroom. Expert speakers 
presented at these events, and participants took part in structured discussions about their 
experiences as they implemented the strategies during the year. 


Within the programme there was scope for each participating school to tailor its approach to 
its own context. The expectation was that participating schools would be at different stages of 
development and have different priorities, and therefore the facility to tailor their use of the 
research evidence to their context was considered essential. 


e School visits by CPD Lead 


Termly visits were made by the CPD Lead to each of the ten participating schools. The aim of 
these visits was to: 


= provide support to the three participating staff, offering advice and guidance, and 
helping to tailor support to individual school context; and 


"observe a lesson putting CPD learning into practice, and offer advice and support 
through the process of Professional Learning Conversations. 


e Ongoing email and phone advice and guidance by CPD Lead 


In addition to CPD days, and visits from the CPD Lead, participating schools were encouraged 
to contact the CPD Lead by email or phone for advice and guidance as the project 
progressed. 


e Classroom implementation 


The project supported participating schools to implement techniques learnt in the CPD 
sessions (such as metacognition and self-regulation, learning culture and feedback) in the 
classroom. Schools were free to select the focus for their school according to the school 
context and school development plan priorities. 


e Collaborative CPD and professional learning conversations 


The pilot also aimed to foster a professional learning culture within participating schools by 
encouraging professional collaboration and learning conversations. The expectation was that 
all three members of staff—the Literacy Subject Lead, the Numeracy Subject Lead and the 
classroom teacher—would attend the CPD events and support classroom implementation 
through regular meetings and lesson observations. 


e Engagement with senior leadership 


2 https://educationendowmentfoundation.org.uk/toolkit/toolkit-a-Z/ 
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To facilitate pilot delivery, the project team put in place a number of measures to maintain the 
engagement of senior leaders within participating schools. Measures included inviting 
headteachers to a launch event, providing guidance on how they could support pilot 
implementation, inviting headteachers to the final CPD session to share learning from the 
pilot, and convening a steering group that included two headteachers from participating 
schools. 


Background evidence 


The gap between research evidence on one hand and policy-making and professional practice on the 
other has increasingly become the focus of attention in the UK as well as across the world (Cooper et 
al., 2009; Nutley et al., 2007). A number of studies have sought to define (research) knowledge 
mobilisation and propose ways for improving its effectiveness in education and overcoming the 
existing barriers (Cooper et al/., 2009; Levin, 2011). 


The barriers to research engagement include skill issues (for example, an inability to interpret 
research findings), resource issues (Such as lack of time or access to academic publications), and 
insufficient rewards in the system (Bransford et al., 2009; Nutley et al., 2007; Sharples, 2013; 
Hemsley-Brown and Sharp, 2003). It has also been noted that research evidence needs to be 
transformed before it can be used in teaching practice: this involves more than simply summarising it 
and requires effective collaboration between teachers and researchers and/or ‘mediation’ (Nelson and 
O’Beirne, 2014). 


Even when schools believe that there is merit in using research to support school improvement, 
teachers can find it difficult to implement general approaches that are evidence-based, such as 
effective feedback and metacognition. The focus of this programme on professional development may 
help teachers to understand and apply specific evidence-informed techniques and interventions in the 
classroom. The Rochdale intervention aimed to communicate research evidence to teachers in a form 
that could be more easily used in their practice (via presentation and workshops at the intervention 
events), as well as building capacity among the teachers to try new approaches with support from the 
CPD Lead. 


Central to model is the idea of a small cluster of schools collaborating with a single research lead. The 
emerging lessons would be applicable across similar formal and informal school networks, including 
teaching school alliances. 


This project is one of the EEF’s Research Use in Schools funded projects. These projects were 
funded to explore knowledge mobilisation in the teaching profession and how research evidence is 
integrated into teachers’ practice and school processes. The projects funded under this round were 
designed to explore three key questions: 


e How can research organisations and others effectively communicate their findings and engage 
with schools? 

e How can schools overcome the barriers to using research well? 

e How can brokers and mediators help schools find and use evidence-based approaches? 


Four other projects were funded as part of the Research Use in Schools funding stream, in addition to 
Research into Practice — Evidence-informed CPD in Rochdale:? 


3 More detail on these projects can be found at https://educationendowmentfoundation.org.uk/projects/projects-a-z/research- 
use-in-schools/ 
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1. Research Champions: a pilot project run by the Ashford Teaching Alliance, testing whether a 
‘research champion’ working across six schools to improve the awareness and use of 
evidence in the classroom is a feasible model. 


2. Research Learning Communities: an efficacy trial of a project developed by the Institute of 
Education to examine whether evidence champions are effective at promoting research use in 
their school when supported by a research community of peers from local schools and an 
academic facilitator. 


3. The Literacy Octopus — Communicating and Engaging with Research: a large multi-arm 
randomised controlled trial investigating a range of different methods of communicating 
research to schools and engaging them in research evidence. 


4. The RISE Project — Evidence-informed school improvement: an efficacy trial of a project led 
by Huntington School that aims to test whether a research-based school improvement model 
makes a significant difference to classroom practice and student outcomes. 


This report presents findings of the first evaluation of the Rochdale Research into Practice 
programme. This is a pilot study, and in the conclusions of the report we comment on whether this 
programme is ready to be evaluated in a trial. 


Evaluation objectives 


The evaluation (like other projects in the EEF’s Research Use in Schools round) had the following key 
objective: 


To explore whether, and to what extent, research communication and engagement strategies 
have the potential to improve teachers’ use of, and attitudes towards, academic research to 
support pupils’ progress. 


Other evaluations in the EEF’s Research Use in Schools round looked at the impact of different 
research communication and engagement strategies on pupil attainment. (One of the EEF’s key 
objectives is to improve pupil attainment.) The original design for the Rochdale evaluation also 
included analysis of pupil attainment data from the National Pupil Database. The intention was to 
measure pupil progress using Key Stage 2 attainment data and to compare pupils in the schools 
participating in the intervention with a matched comparison group from schools that did not take part. 
However, both the EEF and the evaluation team decided that assessing pupil progress at KS2 would 
not be an appropriate measure of the programme’s success, or potential, at this stage in its 
development. In particular, it was felt that pupil progress was a longer-term goal for this particular 
project, rather than a short or medium term one (as set out in the logic model), with one year being 
insufficient to detect impacts on pupils. Also, the process evaluation revealed that the pupils who were 
directly affected by the intervention belonged to year groups that would not be captured in the Key 
Stage 2 attainment data. These two factors meant that any change in pupil attainment identified, 
positive or negative, could not be linked in any way to the programme. Consequently, analysis of pupil 
attainment data was not undertaken and is not reported on in this report. 


Project team 


The intervention was delivered by the CPD Lead, a senior professional development consultant based 
at one of the ten participating primary schools in the Rochdale area. The CPD Lead was supported by 
a steering group and an expert advisor. 
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The independent evaluation team was led by Dr Svetlana Speight at NatCen Social Research who 
was supported by Meg Callanan, Dr Julia Griggs, Dr Javiera Cartagena Farias and Alexandra Fry. The 
evaluation team was responsible for the design and delivery of the evaluation, including a logic model 
workshop, baseline and outcomes surveys of teachers, interviews with staff, observations of training 
events, analysis and reporting. Questionnaires for the surveys were developed by the National 
Foundation for Educational Research (NFER) (Poet et a/., unpublished). The same questionnaires 
were used across all projects funded in the EEF’s Research Use in Schools round. 


Ethics 


The project was reviewed by NatCen’s Research Ethics Committee and received approval in 
September 2014, prior to data collection. 


Headteachers from participating schools signed an annex to the IPLCN partnership agreement that 
included consent to provide the CPD Lead with names and email addresses (and phone numbers 
where appropriate) of all teachers at their school for passing on to the evaluation team (See Appendix 
A). In addition, written information about the study was sent to the headteachers by post prior to the 
baseline survey taking place (see Appendix B), and information for teachers about the surveys was 
sent to them by email (Separately for the baseline and outcomes surveys—see Appendix B). 
Information about process evaluation activities was provided to participants by researchers 
undertaking interviews and observations via email and again face-to-face or on the telephone at the 
time of data collection, and consent was sought prior to interviews and observations taking place. 
Information about the study was also available on the study webpage of the NatCen website, a link to 
which was included in communications with research participants. 
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Methods 


Recruitment 


The ten primary schools participating in the intervention were part of the Inspirational Professional 
Learning Community Network (IPLCN). The network consists of eleven primary schools in the 
Rochdale area, and was established to foster support and partnership across all areas of the school 
curriculum. All the schools in the network participated in the Research into Practice project except one 
that was going through a period of change. The CPD Lead, based at one of the schools in the 
network, was well connected with headteachers at the schools in the network and was able to build on 
that while delivering the Research into Practice programme. 


Each participating school identified three members of staff to participate in the project. These were: 
e the Literacy Subject Lead; 
e the Numeracy Subject Lead; and 
e aclassroom teacher who took the lead role in implementing the project with their class. 


Three members of staff were chosen because the project has a strong collaborative element to its 
approach to professional development. The Literacy and Numeracy Subject Leads were selected 
because it was felt their roles provided the greatest scope for disseminating the learning from the 
project to the wider school. The project also specified that the professional practice of the classroom 
teacher taking the lead on implementation should be judged by the school to be ‘good’ or ‘outstanding’ 
to ensure classroom implementation was as effective as possible. 


The pupils who participated in the intervention were in the classes of the teachers who had been 
selected to lead implementation. There were approximately 30 pupils directly participating in the 
intervention in eight of the schools, and eight to ten children in the other two schools. The age of 
pupils directly involved in the intervention was determined by the year group taught by the nominated 
classroom teacher: in three schools, the focus was on classes in KS1, while the other seven focused 
on children in KS2. 


Data collection 


The evaluation of the Rochdale Research into Practice pilot study included elements of formative 
evaluation, process evaluation, and quantitative data collection and analysis. Findings from the pilot 
evaluation were shared with the delivery team as they became available in order to enable 
collaborative working and facilitate ongoing development of the intervention. 


As part of this evaluation, several methods were used, including: 


e the development of a logic model; 
e apre- and post-intervention survey of teachers; and 
e process evaluation elements, including interviews, observations and case studies. 


Logic model 


A logic model workshop was held with the CPD Lead and another teacher participating in the pilot in 
October 2014 to identify the resources, activities, outputs, and intended outcomes of the programme. 
The logic model was then drafted by NatCen researchers and finalised in consultation with the CPD 
Lead. A copy of the logic model was provided to the project delivery team for sharing with participants 
at the training events (see Logic model section). 
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Surveys 


A key objective of the evaluation was to explore whether the programme demonstrated the potential to 
improve teachers’ use of, and attitudes towards, academic research. This was assessed using 
baseline and outcomes surveys developed by the NFER for the EEF specifically to use in the 
Research Use in Schools round evaluations (Poet et a/., unpublished). 


Surveys of participating teachers were administered online by NatCen Social Research at two time 
points: during the start-up period (baseline) and in the intervention’s final school term (outcomes). All 
teachers at the participating schools were invited to take part in the baseline survey, and those who 
completed the baseline survey were invited to take part in the outcomes survey. 


A letter was initially sent to the headteacher of each participating school in September 2014 to inform 
them of the upcoming survey and to ask for their support in encouraging teachers to complete it. 
Individual teachers were then sent an email, which included a weblink to the survey, followed by a 
maximum of three email reminders. The baseline survey launched during the week commencing the 
15 September 2014 and closed in late October 2014. This meant that the teachers attended the first 
two CPD days during the baseline survey fieldwork period. The outcomes survey launched during the 
first week in June 2015 and closed in late July at the end of the summer term. All teachers who had 
participated in the baseline survey were invited to take part in the outcomes survey via a unique 
weblink sent by email. This was followed by up to four reminder emails. The outcomes survey closed 
at the end of the summer term 2015. 


The survey analysis focused on change over time between the two surveys. There was no comparison 
group, so it was not possible to compare the changes to what would have happened in the absence of 
the intervention. The main outcomes of the intervention were estimated using descriptive analysis 
(frequencies and cross-tabulations) and paired sample t-tests.* Cronbach's alpha was used to assess 
the internal consistency of the outcome measures, and McNemat’s test to explore whether differences 
between responses at baseline and follow-up were statistically significant for individual questions. All 
analysis was conducted in IBM SPSS Statistics for Windows, Version 21.0. 


Process evaluation 


The process evaluation was based on depth interviews, observations of CPD events, and data 
collection in the outcomes survey. Interviews and observations took place throughout the academic 
year in order to capture experiences of participants as the intervention was being implemented. All 
schools took part in at least some process evaluation activities to enable the evaluation to capture a 
full range of perspectives. 


To explore the implementation of the project, nine telephone interviews with the participating 
classroom teachers were carried out in the autumn term of 2014 to gather early feedback. Five of the 
ten participating schools were selected for case study visits in the Summer term 2015. The case study 
schools were selected to reflect the diversity of classroom implementation and included schools 
focusing on implementing strategies in different areas of the curriculum including literacy, numeracy, 
and speaking and listening. These visits involved interviews with the classroom teacher, the Literacy 
Lead and the Numeracy Lead. Observations of a professional learning conversation were also carried 
out in three of the five case study schools. 


Further process evaluation involved three interviews with the CPD lead in the autumn, spring and 
summer terms, two observations of CPD training days, and five follow-up telephone interviews with 
classroom teachers at the five schools that did not participate as case studies. 


“ Measures of effect size were not considered appropriate given the absence of treatment and control groups. 


Education Endowment Foundation 12 


Rochdale Research Into Practice 


2014/2015 Overview of fieldwork carried out 
e Observation of launch event 
Autumn term e Logic model workshop 
e 9 telephone interviews with classroom 
leads 
e 1 telephone interview with CPD lead 
Spring term e 2case study visits 
e 3-case study visits 
Summer term e 2 telephone interviews with CPD lead 
(April and July) 
e 5 telephone interviews with non-case 
study classroom teachers 
e Observation of CPD session 


Interviews were based on a topic guide to ensure systematic coverage of key issues, but were also 
intended to be flexible and interactive, allowing issues of relevance for individual respondents to be 
covered through detailed follow-up questioning. The interviews were digitally recorded and 
subsequently analysed using Framework in NVivo, a systematic approach to qualitative data 
management developed by NatCen Social Research. All collection and analysis of data was 
conducted by the independent NatCen process evaluation team. 


In addition to the qualitative interviews, the outcomes survey included a series of process questions 
which asked those teachers directly engaged with the project to provide feedback on the effectiveness 
of the programme and its different components. Answers to these questions were analysed using IBM 
SPSS Statistics for Windows, Version 21.0. 


Timeline 

Date Activity 

Summer 2014 Schools recruited to the intervention 

September-October |= Programme delivery started 

2014 Baseline survey of teachers, logic model workshop, first 
observation of a training day, and first process evaluation 
interviews 

December 2015 First phase of process evaluation interviews is completed 

April 2015 Second interview with the CPD Lead (out of three) 

May-July 2015 Final stage of process evaluation interviews and observations 

June-July 2015 Outcomes survey of teachers 

August-December Analysis and reporting 

2015 

Costs 


Information on the costs of the intervention was collected from the CPD Lead of the project. Our 
estimate of the cost of a school participating in the Research into Practice programme includes the 
costs of providing the training and the costs for teachers attending the intervention events (there were 
no associated up-front costs). Costs per pupil were calculated using the number of pupils receiving the 
intervention, which is estimated to be around 280 pupils in total across the ten participating schools. 


Education Endowment Foundation 13 


Rochdale Research Into Practice 


Findings 
Participants 


The ten primary schools participating in the intervention are part of the Inspirational Professional 
Learning Community Network. Nine of these were in Rochdale and one in a neighbouring area. The 
size of schools was 368 pupils on average—higher than the national average for primary schools (263 
pupils). Three of the participating schools were smaller than average, and seven were larger. The 
percentage of pupils eligible for free school meals (FSM) was 26.2% on average, about the same as 
the national average of 26.6%.° FSM eligibility varied between the schools from 21% to 31%. 


There were no school drop-outs from the programme over the course of the evaluation, although in 
four of the ten schools there were changes to participating staff because of staff leaving, going on 
secondment, or taking maternity leave. In one case this meant only two teachers participated in the 
project (rather than the three specified by the logic model). In other cases, replacement staff were 
found. 


For the survey element of the evaluation, a// teachers at the participating schools were invited to take 
part, at baseline and at the end of the academic year, as the programme aimed to achieve change at 
the whole-school level rather than only for teachers and pupils directly involved in the programme. 
When reporting the findings of these surveys, we present estimates both for all teachers, and for 
different subgroups depending on their level of involvement. 


The overall response rate for the survey was 95% at baseline (169 of an eligible 177 respondents) and 
73% for the outcomes survey (124 of an eligible 169 respondents). 


Approximately half of those responding to the survey were classroom teachers, a further quarter 
middle leaders, 20% senior leaders, and 5% headteachers. There was considerable variation in 
teaching experience, ranging from NQTs (9%) to those with more than 30 years’ experience (10%). 
More than half of the teachers surveyed had some involvement in the programme, be it first-hand 
(20%) or via colleagues (36%). 


Logic model 


A logic model workshop was held with the CPD Lead and another teacher participating in the pilot who 
participated in the workshop because of their good understanding of the project and its aims. The 
workshop was conducted at the beginning of the evaluation to identify the resources, activities, 
outputs, and intended outcomes of the programme. The logic model was then drafted by NatCen 
researchers and finalised in consultation with the CPD Lead (see Figure 2). A copy of the logic model 
was provided to the project delivery team. The project delivery team shared the model with 
participants at the training events to help clarify the aims and objectives of the project. 


The logic model informed the development of topic guides for the process evaluation, however it was 
not used for construction of data collection instruments for the quantitative element of the evaluation 
(the survey). This is because the survey questionnaires were developed by the NFER to be the same 
for all projects in the Research Use in Schools round. As a result, the outcomes at the teacher and 
school level that were measured in the baseline and outcomes surveys of teachers do not precisely 
match the logic model of this programme. 


Findings from the process evaluation (see section on Process Evaluation Results) indicate that the 
activities set out in the logic model were largely delivered as intended, with good levels of attendance 


° For national statistics, see https://www.gov.uk/government/organisations/ofsted 
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at the seven CPD events, and good engagement with the CPD Lead school visits. In a change to the 
planned activities outlined in the model, participating schools were paired up in the summer term and 
visits to partner schools were carried out to share practice and observe each other’s classroom 
implementation. The activities and outcomes outlined in the logic model (Figure 2) are discussed in 
further detail in the findings from the process evaluation. 
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Figure 2: Rochdale Research into Practice programme: Logic Model 
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Evidence to support theory of change 
Survey findings — outcome measures 
NFER guidance (Poet et a/., unpublished) recommends using six outcome measures: 


positive disposition to academic research informing teaching practice; 

use of academic research to inform selection of teaching approaches; 
perception that academic research is not useful to teaching; 

perception that own school does not encourage use of academic research; 
active engagement with online evidence platforms; and 

research knowledge. 


oa rRaon br 


We could not use outcome measure 6 as the questions were not asked at baseline. The other five 
outcome measures were constructed using relevant survey variables following the NFER guidance 
(see Appendix D for details of the measures). The Cronbach's alpha score, and therefore the reliability 
of the measure, varied considerably between the five scales. The results are displayed in Table 1 
below: 


Table 1: Cronbach’s alpha scores for the five composite measures 


Cronbach’s 
alpha in Reliability of the 
outcomes measure 
survey 


Cronbach's 


Outcome measure alpha in baseline 
survey 


Measure 1: positive disposition to 
academic research informing 0.80 0.85 High 
teaching practice 


Measure 2: use of academic 
research to inform selection of 0.49 0.68 
teaching approaches 


Low at baseline, 
moderate at 
outcomes 


Measure 3: perception that 
academic research is not useful to 0.55 0.65 Low/moderate 
teaching 


Measure 4: perception that own 
school does not encourage use of 0.34 0.35 Very low 
academic research 


Measure 5: active engagement with 


: : 0.60 0.56 Low/moderate 
online evidence platforms 


The Cronbach's alpha scores for measure 4 were considerably lower than for the others, with scores 
so low as to suggest the internal consistency was very poor and the measure was not reliable. For this 
reason measure 4 has not been used in this report. Instead, we analysed the two separate questions it 
was supposed to be based on. 


Each section below looks first at the overall score for each outcome measure (with the exception of 
measure 4) including any change between the baseline and the outcomes surveys. It then considers 
differences between groups of teachers according to their level of involvement in the programme 
(based on the teachers’ own responses to the outcome survey rather than on any pre-allocation to one 
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of the three groups). Finally, this section explores the individual questions that make up each 
composite indicator. 


All analysis has been conducted on data collected from teachers taking part in both the baseline and 
outcomes survey. Where teachers took part in the baseline survey only, their data was not analysed. 


Positive disposition to academic research in informing teaching practice 


Results of the first composite measure showed a statistically significant increase in teachers’ ‘positive 
disposition to academic research in informing teaching practice’, with total mean values increasing 
from 20.6 (SD = 3.3) at baseline to 21.5 (SD = 3.6) in the outcomes survey (t = -2.94, p = 0.004). 


Looking separately at groups of teachers according to their involvement in the project (those involved 
directly, those involved indirectly through colleagues, and those not involved) the survey shows the 
most pronounced change for those with first-hand involvement, the only group for whom results were 
statistically significant (see Table a5 


Table 2: Positive disposition to academic research in informing teaching practice by level of 
involvement 


Baseline - mean | Outcome - mean Observations (N) 


EYefo) c=) score 
First-hand involvement 20.8 22.9% 24 
Colleaguels were involved and 20.9 216 43 
shared learning 
No involvement’ 20.4 20.8 54 
All teachers 20.6 21.5* 124 


* Difference is statistically significant (p < 0.05). 


Note: ‘All teachers’ (N = 124) includes those who did not answer the question about their involvement in the 
programme. 


Looking at each of the individual questions across the baseline and outcomes surveys revealed some 
interesting trends in terms of teachers’ attitudes towards research informing teaching practice. For 
example, 66% of teachers who reported using ‘articles, reports, books or summaries based on 
academic research’ indicated that they found them ‘easy’ or ‘very easy’ to understand at baseline; this 
showed a significant increase to 73% in the outcomes survey.8 Likewise, the proportion of teachers 
who ‘agreed or strongly agreed that that they knew where to find relevant research, were able to relate 
research to their own context, and to use information from research to help implement new 
approaches in the classroom’ also increased significantly between the baseline and outcomes surveys 
(see Table 3). 


5 When the same variables were included in an OLS regression model, with the baseline score as a control 
variable and outcome score at the dependent variable, there was a statistically significant difference by level of 
involvement in the programme. 

’ The group ‘no involvement’ includes those who answered question 13 about the level of involvement with ‘yes, 
my colleagues were involved, but | don’t know any more about it’ or ‘I am not sure’. 

® Frequencies exclude the missing cases and the respondents who said that they ‘did not use this 
source’ (19 cases at baseline and 15 at follow-up). All have been included in the composite outcome 
variable. 
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Proportions also increased for those reporting that research ‘plays an important role in my/our 
teaching practice’, although this change was not statistically significant. In contrast, the proportion of 
teachers who reported feeling confident about analysing information from research fell from 62% at 
baseline to 59% in the outcomes survey. Again, this result was not statistically significant. 


Table 3: Response to individual items included in the question about how much research is 
used at work 
Outcome - 
strongly agree / 
agree (%) 


Baseline — fo) oS-7-1aVF-Vile) atm (iN) 


strongly agree / 
agree (%) 


Plays an important role in my/our 


teaching practice a“ . oe 
Kn her find relevan 
ow where to find relevant 56 72% 122 
research 
Abl I 
ble to relate research to own 71 g2* 121 


context 


Use information from research to 
help implement new approaches 60 71* 122 
in the classroom 


Feels confident about analysing 


information from research 62 59 2) 


* Difference is statistically significant (p < 0.05). 


Use of academic research to inform selection of teaching approaches 


Results of the paired sample t-test for the second outcome measure showed a rise in the total score 
from 4.3 at baseline to 4.6 at follow-up (t = -1.80), but it was not statistically significant (p = 0.07). 


The change in score was largest for those with first hand involvement in the programme, the only 
group for whom results were statistically significant (see Table 4).° 


Table 4: Use of academic research to inform selection of teaching approaches 


Baseline - mean | Outcome - mean 


Observations (N) 


yore) f=) score 
First-hand involvement 4.3 5.0* 24 
Colleague/s were involved and 43 AA 42 
shared learning 

No involvement 4.4 4.5 54 
All teachers 4.3 4.6 120 


* Difference is statistically significant (p < 0.05). 


Exploration of the individual items that comprised the composite measure across the baseline and 
outcomes surveys showed mixed results. While there was a large significant increase in the proportion 


° When this association was explored using OLS regression (as with measure 1), no statistically significant 
difference was found in the outcome score by level of involvement in the programme. 
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of teachers who found ‘articles, reports, books or summaries based on academic research’ important 
in shaping their approach (a rise from 3% to 12%), differences in the other questions were not 
statistically significant. One of the items included in the measure, ‘online evidence platforms or 
databases are important in informing choice of approach’, increased (non-significantly) from 5% to 
12%. Likewise, the proportion of teachers saying that they consulted academic articles, reports, books 
or summaries ‘a lot’ increased (non-significantly) from 12% to 17%. In contrast, the proportion of 
teachers who said that their decision to adopt a specific approach was ‘strongly influenced’ by its 
backing by academic research fell somewhat from 44% to 43% (this difference was not statistically 
significant). 


Perception that academic research is not useful to teaching 


For measure 3 focusing on perceptions of academic research, the total mean score fell from 4.7 (SD = 
1.5) at baseline to 4.3 (SD = 1.4) in the outcomes survey (t = 2.88, p = 0.005), indicating that teachers 
were less likely to report a negative perception of research at the end of the programme (see Table 5). 


All groups of teachers saw the same drop in the score regardless of their level of involvement. 
However, teachers with first-hand involvement began with a lower baseline score (4.2), suggesting 
that teachers selected for the programme had a more positive attitude to research evidence on this 
particular measure.”° 


Table 5: Perception that academic research is not useful to teaching by level of involvement 


Baseline — mean 


Outcome - mean 


Observations (N) 


score score 

First-hand involvement 4.2 3.8 24 
Colleaguels were involved and A8 AA 43 
shared learning 

No involvement 4.8 4.4* 54 
All teachers 4.7 4.3* 124 


* Difference is statistically significant (p < 0.05). 
Note: ‘All teachers’ (N = 124) includes those who did not answer the question about their involvement in the 
programme. 


Each of the two component questions showed a change between the baseline and outcomes survey, 
but neither difference was statistically significant. The proportion of teachers who ‘disagreed’ or 
‘strongly disagreed’ that they ‘do not believe using information from research will improve pupil 
outcomes’ increased from 71% to 79%. The change was smaller in the second item ‘research 
conducted elsewhere is of limited value to school’, where the proportion of teachers who ‘disagreed’ or 
‘strongly disagreed’ rose from 61% to 65%. 


Perception that own school does not encourage use of academic research 


Composite measure 4 ‘perception that own school does not encourage use of academic research’ has 
been excluded from this report on the basis of its poor internal consistency (as discussed above). 
Instead, the evaluation included analysis of the individual questions brought together under this 
measure in the NFER guidance. 


*° As with measures 1 and 2, the association was tested using OLS regression analysis. Results showed no 
statistically significant relationship between the outcome measures and level of involvement in the programme. 
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Comparing baseline and outcome survey responses to each of the component questions shows mixed 
results. The proportion of teachers who ‘disagreed’ or ‘strongly disagreed’ that ‘other staff in my school 
rarely use information from research to inform their teaching practice’ showed a large and statistically 
significant increase (from 31% to 45%), which suggests a positive change. In contrast, the proportion 
of teachers who ‘agreed’ or ‘strongly agreed’ that ‘school leaders do not encourage use of research to 
improve practice’ showed a non-significant increase from 8% to 16% between the baseline and 
outcomes survey. However, the proportion of teachers who ‘disagreed’ or ‘strongly disagreed’ with the 
statement also increased slightly (from 63% to 66%). Results are displayed in Table 6. 


Table 6: Perception that own school does not encourage use of academic research 
Other staff in my school rarely use information from 


research to inform their teaching practice desk hli is Ourcome 5 
Strongly agree 3 2 
Agree 12 11 
Neither agree nor disagree 55 43 
Disagree 27 39 
Strongly disagree 4 6 
Observations (N) 122 122 


My school leaders/governors do not encourage me 
to use information from research to improve my 


practice 

Strongly agree 2 4 
Agree 7 12 
Neither agree nor disagree 29 18 
Disagree 47 36 
Strongly disagree 16 30 
Observations (N) 122 123 


Note: Percentages may not sum to 100 due to rounding. 
Active engagement with online evidence platforms 


The average mean score for all teachers showed very little change over the course of the programme, 
from 4.5 (SD = 1.1) at baseline to 4.6 (SD = 1.0) at follow-up (t = -1.13) (See Table 7). This difference 
was not statistically significant (p = 0.261). 


Exploration of the results by level of involvement in the programme shows little difference between the 
three groups of teachers. None of the differences over time were statistically significant.” 


Table 7. Active engagement with online evidence platforms by level of involvement 


As with the other measures, the association between the outcome measure and level of involvement in the 
programme was explored using OLS regression analysis. Results showed no statistically significant association. 
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Outcome - mean 


Observations (N) 


Xero) g} Xero) ) 
First-hand involvement 4.8 5.0 24 
Colleaguels were involved and 46 46 43 
shared learning 

No involvement 4.3 4.4 54 
All teachers 4.5 4.6 124 


Note: ‘All teachers’ (N = 124) includes those who did not answer the question about their involvement in the 


programme. 


When considered separately, the two component questions showed generally positive trends, but 
differences did not reach conventional levels of statistical significance. While the number of teachers 
who consulted online evidence platforms and databases ‘a lot’ remained stable between the two 
surveys, fewer teachers said that they didn’t consult them at all (a drop from 34% to 29%). As well as 
increased use, teachers were more likely to report finding it ‘very easy’ to understand online evidence 
platforms (an increase from 14% to 17%). However, the proportion of teachers who found online 


platforms difficult to use remained stable across the two surveys at around 28%.” 


= Frequencies exclude missing cases and the respondents who said that they ‘did not use this source’ (34 cases 
at baseline and 30 at follow-up). All have been included in the composite outcome variable (see previous 


footnote). 
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Feasibility 


The Rochdale Research into Practice programme was well received by staff with first-hand 
involvement in the project, with 96% reporting that they would recommend the programme to another 
school. 


This section reports on the feasibility of the project, describing its core elements, summarising 
feedback on each component from the perspective of participants, identifying facilitators and barriers 
to delivery, and reporting formative findings on how the project could be developed in the future. 


Table 8: Overview of participating staffs’ views of Rochdale Research into Practice project 
activities 


AVA=1 aVare(oxeye Mt CZ1)) foltTi t=meforete m CZ1)) Average (%) | 


Research into Practice 67 33 
events 

Research into Practice lead 71 29 
visits 


Collaborative planning and 33 13 4 
feedback 


Individual supportladvice 
from the Research into 
Practice lead via 
phonelemail 


Base = 24. 
Note: The four respondents who did not use individual support/advice from the CPD lead have been excluded 
from percentages reported in the final row. 


Research into Practice CPD events 


Throughout the year, teachers attended three full-day and four half-day CPD events led by the 
Research into Practice lead (detailed in the table below). The events were designed to combine taught 
sessions (Sometimes with expert guest speakers) that introduced the research evidence on 
metacognition, self-regulation, feedback and professional learning conversations; with sessions that 
focused on practical implementation in the classroom. Overall the CPD Lead reported good levels of 
attendance and engagement at events. 
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Timeline of CPD events 
CPD session 


Session 1, September 2014 


(whole-day event) 


Session 2, September 2014 


(whole-day event) 


Session 3, November 2014 


(half-day event) 


Session 4, January 2015 


(whole-day event) 


Session 5, March 2015 


(half-day event) 


Session 6 — April 2015 
(half-day event) 


Session 7 - July 2015 
(half-day event) 
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Content 


Overview of Rochdale Research into Practice intervention 
and its aims 

Keynote speaker on metacognition and self-regulation 
and research evidence on their effectiveness 

How to implement metacognitive and self-regulation 
teaching and learning strategies in the classroom in 
maths and literacy, including ‘talk for learning’ and 
handouts on how to encourage productive ‘group talk’ 


Keynote speakers on developing metacognition in maths 
and literacy from Every Child Counts’ and Reading 
Support 

Tools and resources for classroom implementation 
including approaches to use to support metacognition in 
reading and writing 

Time for schools to plan implementation 


Review and discussion of classroom implementation to 
date 

Focus on developing metacognition and self-regulation in 
guided work 


Presentation on feedback and the work of Shirley Clarke 
on Outstanding Formative Assessment 

Tools and resources for practical implementation of 
feedback strategies in the classroom, including resources 
from Enriching feedback in the primary classroom by 
Shirley Clarke (2003), and Visible Learning for Teachers: 
Maximizing impact on learning by John Hattie (2012) 
Time for schools to plan implementation 


Presentation on ‘Developing Learning Conversations’ 
Keynote speaker from Every Child Counts on 
‘Professional Learning Conversations’ 

Planning for sustainability in individual schools 


Review and discussion of classroom implementation to 
date 
Time for schools to plan implementation 


School presentations on classroom implementation 


The survey found the CPD events were very well received by participating teachers, with two thirds 
(67%) describing the sessions as ‘very good’ and a further third (33%) as ‘quite good’ (see Table 8). 
Particular strengths of the CPD events highlighted in qualitative interviews with participating teachers 
were: 


*8 Further information on Every Child Counts can be found at https://everychildcounts.edgehill.ac.uk/ 


Education Endowment Foundation 25 


Rochdale Research Into Practice 


e Research dissemination and expert speakers 

Teachers reflected that a key barrier to the use of research evidence in the classroom was the 
time required to source, read, and absorb the latest evidence. Providing this evidence during 
CPD sessions was felt to be a time-efficient way for teachers to access this evidence, while also 
providing an opportunity to discuss practical implementation and ask questions of expert 
academics and teachers. Teachers valued the mix of academic speakers who introduced the 
research evidence, alongside experienced teachers who had applied the techniques in the 
classroom and could provide examples of implementation: 


‘I’ve really enjoyed them, and I’ve enjoyed the speakers when they came... it makes you want 
to research and learn yourself to really take your teaching forward, take your learning forward 
to help the children. That’s how | found it anyway, | found it really interesting to listen to what 
they were talking about and how important research is.. and of course research is really good, 
but you just forget it because you kind of get stagnant in your role.. and very busy.. so it’s 
been nice to relight my enthusiastic fire!’ (classroom teacher). 


e Practical implementation focus 
Teachers appreciated the focus on practical implementation in the classroom. They reported that 
this helped them apply the theoretical learning about metacognition to their classroom practice. 
Videos of techniques being used in the classroom and practical examples of materials (for 
example, sentence stems to promote metacognitive talk) were particularly praised, and teachers 
described using these materials in their own practice. 


e Time for planning 
The CPD events were designed to support participating teachers to apply their learning on 
metacognition, self-regulation and feedback in the classroom. Time was set aside during the 
CPD events to give teachers an opportunity to discuss and plan how they would use the 
techniques. Teachers appreciated this time for collaborative planning with their colleagues as 
finding sufficient time for this during the working week was a challenge for many. 


e Networking opportunities 
The CPD events provided opportunities for teachers from the ten participating schools to share 
their experiences and classroom practice with one another. Teachers found this useful and 
described how they had drawn on the experiences of other schools within the network to inform 
their own practice. 


CPD Lead visits to schools 


To support the practical implementation of metacognition, self-regulation and feedback in the 
classroom, the CPD Lead visited each participating school three times during the course of the year. 
These visits took a ‘Lesson Study’ approach and involved a lesson observation followed by a 
professional learning conversation with the three participating teachers. The aim was to support 
classroom implementation as well as foster a professional learning culture through collaborative 
practice and professional learning conversations. 


The survey found a high level of satisfaction with the CPD Lead visits, with 71% of teachers describing 
the visits as ‘very good’ and 29% as ‘quite good’ (See Table 8). Teachers reported finding the CPD 
Lead visits helpful because they: 


e provided an external fresh perspective on their classroom implementation; 
e helped maintain the focus of the pilot and the momentum of implementation; 
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e provided reassurance that they were ‘on the right track’, increasing their confidence in their 
approach; 

e provided suggestions and a steer for how to develop their practice further; and 

e supported the use of professional learning conversations by modelling the approach and 
encouraging teachers to reflect on their practice. 


School visits to share practice 


In an addition to the planned components of the programme (as set out in the logic model activities), 
the CPD Lead also arranged for participating schools to be paired up, arranging exchange visits in the 
summer term. This change was made because the CPD Lead felt schools would benefit from an 
opportunity to see how other teachers had implemented strategies in the classroom. This opportunity 
was viewed very positively by participating teachers who appreciated the opportunity to observe and 
learn from practice in other schools. In one case, for example, a teacher described finding it valuable 
to observe practice in a school which was further ahead than their own because it helped them see 
how they could take their practice forward. In another case, a teacher described how they intended to 
get in touch with their partner school to share their resources after observing a lesson tackling 
metacognition in numeracy. 


Ad-hoc support from CPD Lead 


The CPD lead was also available by phone or email to respond to queries and provide additional 
support. Teachers described the CPD Lead as accessible and helpful and this was appreciated. The 
survey found that 40% of teachers found this individual support ‘very good’, a further 55% described it 
as ‘quite good’, and 5% as ‘average’. 


Classroom implementation 


The ten participating schools each identified one teacher (the ‘class lead’) and _ practical 
implementation was focused on this class. Schools were free to select the focus for their school 
according to the school context and school development plan priorities. Typically over the course of 
the year, schools implemented a number of changes in classroom practice related to the following: 


e Metacognition and self-regulation 
Schools adopted a number of teaching techniques to encourage metacognitive thinking. 
Schools chose to focus on either literacy or numeracy, and examples of techniques used 
included: 
O encouraging metacognitive talk through the use of ‘talk partners’, ‘talk groups’ and 
modelling of metacognitive talk; 
0 providing ‘sentence starters’ and ‘question prompts’ to foster metacognitive thinking 
and talking; 
oO using a ‘reading salad 
skills; and 
0 using ‘thinking out loud’ and ‘thinking time’ techniques. 


= (McGregor, 2007) to develop literal and inferential reading 


e Learning culture 
To improve attitudes to learning and promote a positive learning culture, teachers 
implemented techniques to foster a growth mindset. Approaches included: 


“ This technique involves reading a text aloud and using a series of cards to indicate what the reader 
is doing. Red cards are marked ‘text’ and are used when reading the text. Green cards are marked 
‘thinking’ and are used when the reader thinks out loud about the text. The aim is to create a ‘reading 
salad’ of green and red cards to illustrate that comprehension is about reading and thinking. 
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oO using the concepts of ‘learning powers’ to encourage and foster positive approaches 
to learning—these were promoted through the creation of characters that embodied 
these concepts and through assemblies and classroom displays; and 

0 creating ‘growth mindset’ classroom displays and changing the ways in which children 
were given feedback to value effort rather than results. 


e Feedback 
Drawing on the work of Shirley Clarke on formative assessment (Clarke, 2014), schools used 
a number of techniques to improve feedback, including: 
o developing peer to peer verbal feedback; 
o adapting written feedback using the ‘two stars and a wish’ approach to include a 
metacognitive wish; and 
oO using feedback stickers to encourage pupils to assess their own learning. 


Teachers were positive about their experiences of implementing learning from the project in the 
classroom. The facilitators to effective classroom implementation were: 


e The facility to tailor implementation to individual context 

Teachers observed that the Rochdale Research into Practice model gave them the freedom 
and flexibility to choose which techniques related to metacognition, learning culture and 
feedback they would implement. This flexibility was highlighted as vital for the success of the 
model because schools were able to tailor their approach to the context of their school and the 
specific needs of their pupils: 

‘We [developed our approach] from what we’d been given to suit our children and suit our 
school’s needs. Having that freedom aspect to it worked very well because no one school’s 
setting or needs is the same as another’ (classroom teacher). 


In one school, for example, participating teachers were able to focus on improving 
metacognitive talk, feeding in to a whole-school objective to improve speaking and listening. In 
other schools, the focus initially was on developing a ‘learning culture’ before moving on to 
metacognition and feedback. 


e Sustained support for implementation 
The combination of CPD events and CPD Lead visits to support classroom implementation 
over the course of the year were viewed positively because they provided sustained support. 
Teachers appreciated being able to trial approaches in the classroom, review their success, 
and make changes accordingly. They also valued having the opportunity to meet and discuss 
implementation at the CPD events and share practice. The sustained nature of the support 
over the course of a year was highlighted as a strength of the programme. 


Collaborative CPD and professional learning conversations 


The pilot also aimed to foster a professional learning culture within participating schools by 
encouraging professional collaboration and learning conversations. The expectation (set out in the 
pilot logic model) was that this approach to professional development would improve professional 
collaboration in the short to medium term, ultimately improving professional practice in the long term. 


To facilitate this collaborative approach to CPD, three teachers in each school (the classroom teacher, 
the Literacy Subject Lead, and the Numeracy Subject Lead) took part in the project. The expectation 
was that all three members of staff would attend the CPD events and support classroom 
implementation through regular meetings and lesson observations. This collaborative model was 
viewed by the CPD Lead as essential to effective classroom implementation because learning could 
be shared. This approach was broadly based on ‘Lesson Study’, a form of classroom action research 
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developed in Japan that involves teachers collaboratively planning, teaching, observing, and analysing 
teaching and learning in a series of ‘research lessons’. This collaborative approach to CPD is intended 
to help teachers see aspects of pupil learning through the eyes of others and share their practice 
knowledge (Dudley, 2011). 


Feedback on this collaborative approach from teachers involved in the programme was very positive, 
with 83% describing the collaborative planning and feedback component of the programme as ‘very 
good’, 13% as ‘quite good, and 4% as ‘average’ (See Table 8). The collaborative approach was felt to: 


e support the generation of ideas because staff could reflect on implementation and share views 
in a non-judgmental way; 

e provide opportunities for staff to observe each other’s lessons and learn from each other’s 
teaching techniques and approaches; 

e support implementation by sharing the workload; 

e increase the likelihood that learning from the pilot would be sustained long-term because the 
project was less vulnerable to staff changes; and 

e increase the likelihood that learning from the pilot would be sustained long term because the 
Literacy and Numeracy Subject Leads would be in a position to implement strategies at a 
whole-school level if they were found to be successful. 


However, some participating schools found it challenging to fully implement this collaborative 
approach and there was variation in the extent to which staff met regularly to plan, implement, and 
review the changes they were making in the classroom. In some instances, this meant collaboration 
was limited to the occasions when the CPD Lead visited the school (or during attendance at CPD 
events), with implementation largely the responsibility of the classroom teacher at other times. The 
barriers to effective collaboration were: 


e a lack of time to meet regularly because of teaching timetables and other responsibilities 
(particularly in the case of senior staff); 

e staff not accustomed or comfortable with this form of collaborative working; and 

e staff changes (such as staff leaving or taking on new responsibilities) leading to disruption 
within project teams (this occurred in four of the ten participating schools). 


In contrast, where this form of collaborative CPD worked well, teachers identified the following 
facilitators that supported implementation: 


e time provided ‘off-timetable’ to meet regularly to plan, implement, and review; and 
e a school culture that already supported collaborative approaches to lesson planning and 
teaching improvement. 


Engagement with senior leadership 


To facilitate pilot delivery, the project team put in place a number of measures to maintain the 
engagement of senior leaders within participating schools. This was viewed as critical to the success 
of the pilot to maintain momentum (for example, by providing time ‘off timetable’ for participating staff 
to meet regularly) and to increase the likelihood that learning from the pilot would be sustained in the 
longer term. Measures taken to encourage this engagement included: 


e inviting headteachers to the launch event so they were clear on the aims and objectives of the 
pilot (a guidance document was produced outlining how senior leaders could support pilot 
implementation and this was discussed at the launch); 
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e inviting headteachers to the final CPD session to share how the pilot had been put into 
practice, to encourage wider learning from the pilot; and 


e convening a steering group that included an expert external advisor and two headteachers 
from participating schools: this was viewed by the CPD lead as instrumental in supporting 
delivery and maintaining senior level engagement with the pilot across the participating 
schools. 


Formative findings 


This section summarises the learning from the individual components of the project to support future 
development of the programme. It draws on suggestions made by the participating teachers and the 
reflections of the CPD Lead. 


CPD events 


e It was suggested that all CPD events should be whole-day events. Some schools felt that 
whole-day events would allow for more networking and sharing of practice, making it easier for 
teaching staff to focus on the training. 


e Consider re-ordering the CPD events so that the session on Professional Learning 
Conversations is earlier in the programme. 


CPD Lead school visits 


e For schools that found it difficult to maintain a collaborative approach to implementation, the 
visits from the CPD Lead were a catalyst to moving the project forward. In these 
circumstances, the suggestion was made to increase the number of visits to one each half- 
term. Where schools were finding it easier to work collaboratively and drive the project forward 
independently, the termly visits were felt to be sufficient. 


Collaborative approach to CPD 


e Explore ways to ensure participating staff are given regular dedicated time to meet to 
collaboratively plan, implement, and review changes in classroom practice: limited time for this 
was viewed as a barrier to successful implementation. Suggestions included: 


oO reducing the number of participating staff from three to two, to make the collaborative 
approach easier to implement; 


0 selecting staff teaching the same year group (if two-form entry), or within the same 
Key Stage, to capitalise on existing collaboration and shared practice 


Although not in all cases, some schools found it harder to work effectively if staff 
participating in the project worked across different key stage groups. However, schools 
and the CPD Lead also reflected that it was of value to have a member of the Senior 
Leadership Team participating in the project as this was felt to increase the likelihood that 
learning from the project would be shared and taken forward across the school. 


Classroom implementation 


e Continue to provide practical examples and materials that can be used in the classroom to 
facilitate classroom implementation. Videos of good practice were particularly appreciated. 
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e Continue to allow flexibility for schools to tailor the strategies they use to their school context 
as this was viewed as key to ensuring engagement and buy-in from teachers and senior 
leadership teams. 


Perceived outcomes 


This section reports on the perceived impacts of the pilot. It draws on findings from the survey and 
qualitative interviews with participating staff. 


Ninety-one percent of staff with direct or indirect involvement in the project (n = 67) felt that the 
programme provided them with the right amount of information, and that as a result they were able to 
enhance their practice in a variety of ways (as shown in Table 9). 


Table 9: The extent to which the information provided by the Rochdale Research into Practice 
model helped participating staff 


Rochdale RiP model helped A lot (%) A little (%) No) a= ir 1 20) 
me to: 


Discuss best practice with 70 30 

colleagues in my school 

Share the learning with 39 40 21 
people or organisations 

outside my school 


Conduct my own research or 18 36 46 
enquiry 


Influence colleagues in my 39 40 21 
school to change their 

classroom practice 

Improve my knowledge of a 39 46 15 
topic or subject 


Base = 67. 


Use of research evidence 


Participating teachers reflected on the challenges they face in making use of research evidence in 
their professional practice. Time pressures, challenges in identifying relevant research evidence, and 
taking theoretical research evidence and adapting it for practical implementation in the classroom 
were all issues raised as barriers to effective use of research evidence. Teachers also observed that it 
was critical to have Senior Leadership engagement with research evidence, particularly at the point of 
moving beyond individual teacher implementation to wider school roll-out and whole-school change. 


As set out in the logic model, by the end of the pilot year, a key aim of the project was to have 
improved the use of research evidence by increasing teacher understanding of research evidence and 
its implementation in the classroom, and by helping teachers to develop strategies to critically 
appraise evidence. Participating teachers felt the project had achieved this by providing the time and 
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support to engage with the research evidence in an accessible way and trial putting it into practice in 
the classroom: 


‘| think it’s given us the opportunity to actually go and listen to the research and be able to take 
the time and look into the research. Teachers are busy all the time and they've already got a 
big work load and if they get an email about some new research it’s very unlikely that they 
would download it and read it because there are that many other things to do. Whereas if you 
get given time at a CPD day and you get this research given to you, and you get people 
explaining it, you’re going to take it on board because you've heard it, you understand it, 
you've seen it in action...so that’s been important’ (Numeracy Lead). 


As discussed earlier in the survey findings, the composite measure indicating a positive disposition to 
academic research informing teaching practice showed a statistically significant improvement for 
teachers directly involved in the project (see Table 1), indicating that the programme has achieved the 
aim of increasing teacher understanding of research and its use in the classroom. 


Beyond the research evidence disseminated as part of the programme, 63% of teachers with direct or 
indirect involvement in the project (n = 67) reported seeking out further evidence, research, or 
information about a topic relevant to their practice, and 45% had sought information on other topics or 
approaches. Examples of greater use of research evidence included teachers using the EEF toolkit 
and carrying out their own independent investigations of research evidence. However, this was not 
consistently the case, and other teachers acknowledged that their engagement with research 
evidence was limited to that presented as part of the project, citing lack of time and the challenge of 
taking evidence and implementing it in practice as the primary barriers. These findings suggest that a 
structured programme like that offered provides the greatest scope to embed the use of research 
evidence into practice because it overcomes such barriers. They also suggest that continued use of 
research evidence will require such structures to remain in place so that teachers are supported to 
engage with research evidence in the long-term. 


Teaching practice 


As set out in the logic model, the project aimed to improve teaching practice by helping teachers 
implement metacognition, self-regulation and feedback in the classroom, as well as by supporting 
teachers to establish a professional learning culture through collaboration and professional learning 
conversations. Findings from the survey indicate that the project did have a perceived positive impact 
on teaching practice: 60% of teachers (with direct or indirect involvement in the project, n = 67) felt 
that the project had enabled them to reflect on their own practice ‘a lot’, while a further 37% felt it had 
helped ‘a little’. The survey also found that 70% felt the project had enabled them to discuss best 
practice with their colleagues ‘a lot’, while 48% felt it had helped them change their classroom practice 
‘a lot’ (Table 8). 


Participating teachers expanded on these findings during interviews reflecting on a range of impacts 
on their teaching practice, including: 


e Increased awareness of their teaching approach 
Teachers spoke of having an increased awareness of how they were teaching, and reported 
thinking more explicitly about their approach and its impact on pupils. 


e Increased confidence in teaching techniques to support metacognition 
Teachers described feeling more confident in techniques and strategies they could use to 
support metacognition. They also particularly valued the licence and freedom the project gave 
them to experiment and trial new approaches in the classroom: 
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‘It’s certainly given me a lot of inspiration... | think it’s allowed me to be experimental and try 
things... really having the freedom to go forward and try out some new ideas which have been 
a great success, that has been inspiring. I’ve really enjoyed doing it... It’s almost like I’ve got 
extra wheels in the machine which make lessons run much better for me, with better 
outcomes’ (classroom teacher). 


Pupil impacts 


As set out in the logic model, by the end of the year the project aimed to achieve four short-term 
outcomes: (1) a positive attitude to learning, (2) pupils better able to talk about their learning, (3) pupils 
more aware of what they know and what they need to do next, and (4) pupils beginning to acquire the 
tools to help them plan, monitor and review their (and peer) learning. In the medium term (one to two 
years from the start of the intervention) the expectation was that these outcomes would lead to pupils 
being more active learners, implementing metacognition and self-regulation tools, with the ultimate 
aim of improving literacy and numeracy outcomes in the longer term. 


e Metacognition 


Schools that had focused on improving metacognitive talk reflected positively on the impacts 
on pupils, observing that they had seen an improvement in the quality of the talk children 
engaged in. Modelling productive talk, providing sentence stems to support metacognitive talk, 
working on using ‘talk partners’ and ‘talk groups’, and allocating specific roles within 
discussions to encourage both listening and speaking were all identified as strategies that had 
supported this: 


‘| think one of the biggest impacts is the way the children speak to each other... and the 
productiveness of that. A real positive impact is more productive talk and you can hear those 
conversations going on now with the children, and the children are now actually taking 
ownership and asking the other children “So what do you think? Do you have anything to 
add?” which has been really good in such a short space of time’ (classroom teacher). 


Schools that had focused on metacognition in literacy also reflected on positive impacts, 
describing improvements in comprehension through the use of tools such as the Reading 
Salad (McGregor, 2007). Where schools had focused on metacognition linked to problem- 
solving in maths, positive impacts included pupils increasingly being able to describe and 
explain how they solved problems. 


Both participating teachers and the CPD Lead viewed these as positive improvements in 
metacognitive thinking, but it was also acknowledged that such improvements take time to 
develop and embed and consequently metacognitive teaching strategies need to be 
maintained and sustained over a long period before improvements in pupil attainment data are 
likely to be seen. 


e Positive learning culture 


Schools that had focused on encouraging a positive learning culture, using the concepts of 
‘growth mindset’ and ‘learning powers’, were positive about the impacts of these approaches 
on the pupils’ attitude to learning. In particular, the way in which these approaches 
encouraged a ‘have a go’ attitude, and took the emphasis away from getting the correct 
answers to making the most effort: 


‘The majority of the children will have a go at anything now and they will say “Oh I’m 
challenging my brain... my brain will grow if | challenge my brain”, so | think that kind of ethos 
in the classroom has changed and if they do make mistakes, they say “It’s ok though isn’t it, to 
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make mistakes?”, so | think the ethos of the classroom—that safety net in the classroom that it 
is OK to make mistakes—is there’ (classroom teacher). 


Establishing this culture was seen as an important step in creating an environment in which 
pupils could engage with metacognitive strategies and take these forward. 


e = Self-regulation and feedback 
Linked to changes in learning culture, teachers described improvements in the ability of 
children to work independently and seek support from peers before asking for help from the 
teacher: 


‘We had one or two children who would come up to you after every piece of work and ask “Is 
this right?”... whereas now we don't. Now you can see them chatting to each other... “Is yours 
this? Is yours that?”, which is a big improvement because they’re not depending on me as 
much, they're actually supporting each other on their tables... it’s definitely an improvement’ 
(classroom teacher). 


Where schools had implemented strategies to improve feedback, teachers also spoke 
positively of improvements, with pupils more able to identify where they needed to improve 
and what their next steps were. 


Readiness for trial 


Before the project is ready to be evaluated in a trial, further consideration should be given to the 
following issues: 


Trial focus 


While this intervention was quite well specified, its structure and content need further prescription for it 
to be implemented in a trial in a consistent way. Consideration should be given to the following 
issues: 


Refining project content 


e Which elements of the programme are considered essential to its efficacy, and which 
elements have flexibility—such as the number of CPD events, the number of school visits by 
the CPD Lead, and the level of collaboration expected between participating teachers within 
each school. 


e In its current form, the programme has been piloted as a model for disseminating research 
evidence and supporting the classroom implementation of teaching and learning strategies on 
metacognition, self-regulation and feedback. Further consideration should be given to whether 
the model would be equally effective if different research topics were covered. In other words, 
whether it is solely the structure for research dissemination developed by the Rochdale project 
that would be the focus of a trial, or whether it would be both structure and content. If it is the 
former, the project is reasonably well specified; if the latter, further prescription of project 
content would be required to ensure fidelity to the model and consistency across the trial. 


e Formative findings from the process evaluation suggest that the programme might be more 
effective if a number of amendments were made to its activities, including, for example, 
making CPD training days whole-day events, and finding ways to ensure that staff have 
sufficient time to participate. 
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Replicating the pilot in different contexts 

e This project was implemented within an existing network of schools. Alongside measures 
taken by the project team to engage senior leaders with the pilot, participating staff and the 
CPD lead felt the network facilitated project implementation because of pre-existing 
relationships and structures that fostered partnership working and encouraged buy-in for the 
project from headteachers. Further consideration should be given to the how to replicate this 
level of engagement in schools without a pre-existing network, and what implications this may 
have for trial recruitment and ongoing engagement with the project. 


Defining the treatment group and outcome measures 


e Given that a significant focus of the programme is on improving teachers’ attitudes and 
behaviours as a first step towards improving pupil attainment, it would be important for a trial 
to collect outcome data at teacher level as well as at pupil level. Outcomes at teacher level 
should reflect the logic model: this was not the case in this evaluation since outcomes were 
developed by an external organisation (NFER) and were not tailored specifically to the 
Rochdale Research into Practice model.”° 


e For a trial, pupil-level outcome data would need to be collected: this was not part of this 
evaluation. This would require careful consideration of the year groups that should take part 
and what outcome measures would be appropriate (for example, attainment data only or 
attitudinal data as well). More conceptual thinking is also required to articulate how/if the 
programme intends to impact the whole school—or maybe only pupils in particular year 
groups—bearing in mind that only a certain number of teachers can attend training events and 
have direct involvement in the programme, as this will have a direct impact on how the 
treatment group is defined in a trial. 


Trial length 


e As the logic model indicates, the project team anticipated some short term outcomes at the 
end of the first year of implementation, but longer term impacts (particularly on pupil 
attainment) were not anticipated until two years post-implementation. A trial would need to 
factor in this timeframe and collect outcome data at appropriate time points. 


Cost 
Cost of implementing the intervention 


The majority of the intervention’s direct costs were associated with hiring the venue (including 
refreshments) and speaker fees. Venue costs for all events were £4,500, and speaker costs (including 
fees, accommodation and transportation) were £4,200. 


The salary cost for the CPD Lead was estimated as £18,000 for 40 days of work over 12 months (this 
figure does not include any potential additional or unbudgeted time spent by the CPD lead) and the 
CPD Lead travelling costs were estimated as £54 for the whole intervention length. In addition, school 
administrative assistant work was required, estimated at £95 for the project. There were no financial 
costs associated with teachers using information from the training events, however we have included 
a time cost associated with the additional planning time needed for teachers to implement what they 
had learned. Time required varied from school to school, but on average most schools planned once 
every half-term for two hours. 


*S The reason the outcomes survey was not tailored to the Rochdale model was in order to make the results 
comparable across all EEF projects in the ‘Research Use in Schools’ round. 
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Purchasing (or printing and photocopying) resources and materials associated with the seven events 
was estimated to cost £550 for the year (See Table 10 for more details). 


Table 10: Cost of providing the intervention (financial marginal cost) 


Time (days, 1 day =5 


hrs) Cost (£) Cost (£) per pupil 
Speaker fees (including 
accommodation and transport) ae = 
CPD Lead salary (12 months) 40 18,000 64.29 
Travel costs CPD Lead 54 0.19 
Dissemination meetings 72 0 0 
Planning meetings 180 0 0 
Conference & refreshments 4,500 16.07 
Administration assistant 95 0.34 
Delivery materials 550 1.96 
- I f idi h 
Sub-tota cost or providing t e 292 27,399 97.85 
intervention 


Base: 4,851 pupils are receiving the effect of the intervention. 
Cost and time of attending the events 


The seven events involved teachers travelling to the venue but as travel costs were not reimbursed 
they are not included in the calculations below. 


The major cost associated with staff attending the CPD events related to the provision of staff cover 
for a total of five and a half days (four full-day events and three half-day events). During the pilot, 50% 
of these costs were met by the project while participating schools funded the remaining 50%. In 
addition, the school headteacher attended one full-day and one half-day meeting (1.5 days in total). In 
total, the cost was approximately £4,736 per school. Although only 50% of this was paid by schools, 
the full cost has been included here (see Table 11). 


Table 11: Cost of attending the programme events 


hime (Geyer t dayi= > (oxel-) a (9) Cost (£) per pupil 


hrs.) 


Sub-total cost of attending the 


training (staff cover) 180 eee aioe 


Base: 4,851 pupils are receiving the effect of the intervention. 


Total cost and cost per pupil 


The number of pupils being directly taught by the Classroom Teacher participating in the intervention 
was one class of about 30 children in eight of the schools and from 8-10 children per school in the 
other two schools. Table 12 presents details of the total cost of the intervention and its respective cost 
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per pupil (assuming that 280 pupils in total were affected by the intervention in its first year). Overall, 
the cost per pupil is relatively low at £267 per pupil per year.*© 


Table 12. Total cost of the intervention and cost per pupil 


(orol-) mi (9) Cost (£) per pupil 


Running costs per year 74,759 267 


Total cost in first year 74,759 267 


There were no start-up costs (only one-off costs) associated with this intervention. Therefore, every 
year, independently of how many years the programme runs, the average cost per pupil will be equal 
to the running costs. However, we believe it is important to consider that every year more pupils will 
receive the direct effect of the intervention: we estimate that in the second year 60 pupils in each 
school will benefit (half of those will receive it from the teacher who participated in the intervention the 
previous year, and the other half will receive it from the teacher currently involved). A similar number 
of pupils would receive the intervention in the third year (600 in total). Table 13 shows how the 
average cost per pupil per year decreases as the length of the intervention increases, being around 
£172 per pupil in the third year. 


Table 13. Total cost per pupil over three years of the programme 


Cumulative cost Average cost (£) per 
per pupil (£) pupil per year 
Year1 267 267 
Year 2 392 196 
Year 3 516 172 


The programme is associated with relatively high costs as the number of pupils receiving the 
intervention is not very large. 


16 Assessed as low cost based on the Education Endowment Foundation’s _ toolkit: 


https://educationendowmentfoundation.org.uk/evidence/about-the-toolkits/ 
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Conclusion 


This report presented findings of a mixed-method study evaluating the Research into Practice model 
in Rochdale as it was piloted in the academic year 2014/2015. This intervention consisted of a 
programme of CPD events and activities implemented within a pre-existing network of primary schools 
and was led by a CPD Lead. The focus of the programme in the short term was on teacher-level 
outcomes such as attitudes towards research and the use of research evidence in practice. Impact on 
pupil attainment was viewed as a longer term outcome and outside the timeline of this evaluation. 


Evidence of promise 


There were some improvements in teachers’ attitudes towards research between baseline and follow- 
up. There were statistically significant changes in two of the four outcome measures: 


e an increase relating to outcome measure 1: ‘positive disposition to academic research in 
informing teaching practice’; and 

e a reduction in outcome measure 3: ‘perception that academic research is not useful to 
teaching’. 


There were no statistically significant changes relating to: 


e outcome measure 2: ‘use of academic research to inform selection of academic approaches’ 
(although there was a statistically significant positive change for teachers with direct 
involvement in the programme); or 

e outcome measure 5: ‘active engagement with online evidence platforms’. 


Although our analysis showed improvements in teachers’ perceptions and attitudes, these cannot be 
attributed confidently to the programme in the absence of a counterfactual, however there was some 
indication that teachers with first-hand involvement in the programme saw greater improvements. For 
example, teachers who were more involved in the programme showed a larger positive change in their 
disposition to academic research than those with lower levels of involvement. This suggests that the 
improvements observed were likely to be linked with the programme. 


Findings from qualitative interviews with participating staff found that some were making use of 
research evidence independently of the project, while others reported that their engagement with 
research evidence was limited to that presented as part of the project, citing lack of time and the 
challenge of taking evidence and implementing it in practice as the primary barriers. These findings 
suggest that a structured programme like that offered by the Rochdale Research into Practice project 
provides scope to embed the use of research evidence into practice because it overcomes barriers 
related to time and practical implementation. The findings also suggest that continued use of research 
evidence will require such structures to remain in place to continue to support teachers to engage with 
research evidence in the long-term. 


Was the approach feasible? 


The programme largely ran as intended and documented in the logic model. It was perceived very 
positively by participating staff. There was a substantial degree of buy-in from senior leadership teams 
at the participating schools which was helpful to the intervention. This was facilitated by measures 
taken by the project team to engage with senior leaders, and helped by the fact that the schools were 
members of a pre-existing network. 
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For the programme to be implemented outside of an existing network of schools, careful thinking will 
be needed around how to replicate the level of school engagement achieved in the pilot, as well as 
ensuring that participating teachers feel supported by leadership teams at their respective schools. 


Finding time for working collaboratively on trying to implement research evidence in practice was 
mentioned by participants as a challenge, but overall the requirements of the programme were 
feasible. 


Is the approach ready to be evaluated in a trial? 


We believe that the Rochdale Research into Practice model in its current state of development is not 
ready to be evaluated in a trial. Before a trial is considered, further thought should be given to (a) 
which elements of the project are considered essential for its efficacy, and (b) whether the focus of a 
trial would be to test the project structure as a model for research dissemination, or both the structure 
and content of the project as piloted. The feasibility of implementing the model in schools without a 
pre-existing network would also need to be considered, alongside further clarification of the treatment 
group, outcome measures, and trial length. 


Limitations of the evaluation 
There were a number of limitations to this evaluation of the Rochdale Research into Practice model: 


e The outcomes collected in the teachers’ survey were not wholly consistent with those outlined 
in the intervention’s logic model because the survey was developed to be consistent across all 
projects in the Research Use in Schools round: it was not designed specifically for this pilot. 

e There was no comparison group. 

e Response rates to the teachers’ surveys were exceptionally high at baseline (95%) but were 
lower at follow-up (73%). It is possible that teachers who took part in the follow-up survey 
were systematically different from those who did not take part (perhaps, for example, holding 
more positive attitudes towards research, or being less busy). 

e The sample size available for analysis of outcomes (n = 124) was low, and the differences 
needed to be very large in order to be statistically significant. 

e The evaluation was not able to include analysis of impact on pupil attainment. 


Future research 


Future research opportunities could include a synthesis of findings from projects funded as part of the 
EEF’s Research Use in Schools round. This could help by identifying the most effective strategies for 
engaging teachers with research evidence. Such learning could be beneficial to the Rochdale 
Research into Practice model as it is developed further. 
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Appendix A: IPLCN partnership agreement outlining 
schools’ responsibilities in relation to the Research into 
Practice programme 


Inspirational Professional Learning Community Network 
Research into Practice Programme 
Supported by the Education Endowment Foundation 
Partnership agreement September 2014 - J uly 2015 


Schools will be supported to implement the Research into Practice Programme, which will be 
developed to focus on the application of the strategies for learning and teaching shown by research as 
having the most impact on children’s progress. The work in schools will be undertaken by subject 
leaders for English and mathematics, working with a nominated class teacher. 


bYor aTote) mexe)inlagiicialevals IPCLN commitment 


e identify a member of the leadership team to e provide funding to support the 
take overall responsibility for the success of implementation of the IPLCN Lesson Study 
the initiative in school and to ensure the programme in the school 
completion of all required reporting and 
dissemination e provide access to leading researcher in the 

field of metacognition, self regulation and 

e identify the relevant class teacher to feedback on learning 
undertake the incorporation of the research 
into practical teaching and learning strategies e provide expert professional development in 
English and mathematics the practical pedagogical strategies 


associated with research findings 
e arrange for the close involvement of the 


subject leaders for English and mathematics e fully fund the costs of the Research into 
Practice Lead 


e ensure that all the participating teachers are 


released in order to attend CPD events ¢ provide a dedicated source of programme 
provided by the IPLCN support to and communication with 
participating schools through the Research 

e ensure that all the participating teachers are into Practice Lead 
released in order to undertake the 
observation and debriefing involved in the e provide school site visits and facilitate small 
programme group networks in order to : 

e use the core funding provided by the - support subject leaders and class 
Education Endowment Foundation to support teachers in the applying of research 
the implementation of the Research into into practice in the classroom, 

Practice programme and not for any other 


- obtain a qualitative view of the project 


urpose ie? 
RP in practice in schools; 


e contribute resources from within the school 


ht ie - gather case study material to contribute 
budget to support the release of participating 
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teachers in order to supplement the core 
funding provided by the Education 
Endowment Foundation 


support school site visits by the Research 
Into Practice Lead assigned to the project 
and, where applicable, by members of the 
independent evaluation team 


ensure participation in surveys, interviews etc 
as required by the independent evaluators 


share teacher assessments and any other 
available assessments 


to evidence of the impact of the 
programme on teacher learning and 
pupil progress. 


fully fund the costs of training venues 


work with the independent evaluators to 
gather and analyse data and report on the 
impact of the project 


liaise with relevant personnel in the 
Education Endowment Foundation , to keep 
them informed of the progress of the 
project, any issues arising and to invite 
them to relevant meetings 


convene a steering group to manage the 
project and provide support and challenge 
for the implementation of the programme 
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Inspirational Professional Learning Community Network 
Research into Practice Programme 


Supported by the Education Endowment Foundation 


Partnership agreement September 2014 - J uly 2015 


1. Having considered the respective commitments set out in the partnership agreement 
provided, | confirm that this school wishes to participate in the EEF funded Research into 
Practice programme 2013-15. 


2. Having read the Project Delivery Plan and understood the commitment to releasing 
teachers for CPD events and school visits by the Research Into Practice Lead, | wish to 
register the following teachers for participation: 


Subject Leader 1 
ROM @siccss dee ccwes seduce seaseeteccrerensessstenevars =: NAIM Coie ce ieceeciuccsveveevensesysscseessaeseaesaseueeestees see 
Subject Leader 2 
ROC oie canices saececcntcseciwaesecveteesecesacteenas ee NAN Coir csiiiestcnnescsnice seuiven aie seeedsnaeesmesvnnievarsies ac 
Class Teacher 
ROMO. ese veces tence de vachieaedsaiedes Rew seevedendes NAM Cite csescci sledectinesihacuaiindscestaeceateaseniteustantee 
Signature Job title Ted stole) (folcer-Ualtrclael a) Date 
Headteacher 
IPLCN Lead IPLCN 
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Appendix B: Communications with school staff 


Letter to headteachers: Baseline survey 


Nat Cen ff veoow. NatCen.ac.uk 


W anatcen 


Social Research that works for society F inatcenSociaiResearch 


| Name 
Address 
(Rochdale 
Lancashire) 


11" September 2014 


Dear Name, 


Research-informed schools evaluation: Development of the 
Rochdale Research into Practice model 


| am writing to you to let you know that over the next few weeks teachers in your school will be 
taking part in an online survey for the Research-informed schools evaluation: Development of 
the Rochdale Research into Practice model. 


This survey will help us to understand teachers’ views and experiences of using research in their 
teaching. The study is funded by the Education Endowment Foundation. All information 
collected in this survey will be confidential and individuals will not be identified in the published 
results. 


We would appreciate your support in encouraging all teachers to complete the survey. 
Further information about the study can be found at www.natcen.ac.uk/research-use. If you 


have any queries, please feel free to email researchuse@natcen.ac.uk or call us on 0800 652 
0401. 


J Speepf— 


Dr Svetlana Speight 
Research Project Director 
NatCen Social Research 
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Email invitation to teachers: Baseline survey 


To [firstname lastname], 


| am writing to invite you to take part in a short piece of research for Research-informed schools 
evaluation: Development of the Rochdale Research into Practice model. 


As you may be aware, as part of this programme, teachers in your school will be involved in learning 
how they may be able to enhance your school’s teaching using research-based evidence. 


As part of this process we would like you to answer a few questions about your own teaching 
practice. The online questionnaire should take no longer than 15 minutes. We would be grateful if 
you could complete it by 13th October. 


Take part now 


The information you provide will help us to understand teachers’ views and experiences of using 
research in their teaching. The study is funded by the Education Endowment Foundation. All 
information collected in this survey will be confidential and individuals will not be identified in the 
published results. 


Further information about the study can be found at http://www.natcen.ac.uk/research-use. If you 
have any queries, please feel free to email researchuse@natcen.ac.uk or call us on 0800 652 0401. 


Kind regards, 


J Spetpp— 


Dr Svetlana Speight 
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Research Project Director 


NatCen Social Research 
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Email invitation to teachers: Outcomes survey 


To [firstname lastname], 


Last year you kindly completed a short survey as part of the Research-informed schools evaluation: 
Development of the Rochdale Research into Practice model. 


As part of this programme, teachers in your school are involved in learning how they may be able to 
enhance your school’s teaching using research-based evidence. 


We would like you to answer a few questions about your own teaching practice. The online 
questionnaire should take no longer than 10-15 minutes. We would be grateful if you could complete 
the survey as soon as possible. 


Take part now [unique link] 


Your access code is : [access code] 


The information you provide will help us to understand teachers’ views and experiences of using 
research in their teaching. The study is funded by the Education Endowment Foundation. All 
information collected in this survey will be confidential and individuals will not be identified in the 
published results. 


Further information about the study can be found at http://www.natcen.ac.uk/research-use. If you 
have any queries, please feel free to email researchuse@natcen.ac.uk or call us on 0800 652 9294. 


Kind regards, 
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J Speipf— 


Dr Svetlana Speight 
Research Project Director 


NatCen Social Research 
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Appendix C: The Outcomes Survey 


Supporting Pupil Progress 


Thank you very much for taking part in this survey. Your responses will contribute to a study 
conducted by NatCen Social Research on behalf of the Education Endowment Foundation. It is 
exploring different approaches to improve pupil progress. The survey includes questions on how 
you/your school have decided to introduce new approaches and the types of information you use to 
inform decisions on teaching and learning. 


The survey should take no more than 15 minutes to complete. 


Your answers will be treated confidentially, which means that you and your school will not be identified 
in any reports produced from this research. 


BLOCK A - Introduction 
Job 


1. What is your job role? (Please tick one box below that best describes your role) 


Classroom teacher 


Middle leader (e.g. head of department, subject or curriculum area leader, key 
stage leader, pastoral services leader) 


Senior leader (e.g. deputy or assistant headteacher) 


Headteacher, principal or director 


[| ] ee 


Other role (please specify) 
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HowLong2 


2. How long have you been in the teaching profession? (Please tick the box that describes 
the length of your whole teaching career, including career breaks) 


30 years or more [| 5-9 years [| 
20-29 years [| 1-4 years [| 
10-19 years [| First year of teaching (NQT) [ ] 


BLOCK B - About a specific approach to supporting pupils’ progress 


3a. ApproachO 


Please name in the box below a specific approach that you have used within the last two 
years to support pupils’ progress. For example this could be a teaching method, or a 
resource, product or initiative. 


Name/brief description (please write in the box below) 


Education Endowment Foundation 50 


Rochdale Research Into Practice 


Identify3 


3b. Which, if any, of the following were important in identifying the approach you named? 


Please select the three most important sources. (Please tick up to three) 


Ideas generated by me or my school 
Ideas from other schools 
Advice from my local authority or academy chain 


Articles, reports, books or summaries based on academic research (paper or web 
based) 


Articles, reports, books or summaries based on teacher experience (paper or web 
based) 


The promotional materials of an external supplier 
Action research conducted by me or my colleagues 
Information gathered through training/CPD 


Online evidence platforms or databases (e.g. the Sutton Trust Teaching and 
Learning Toolkit) 


Guidance from official bodies such as DfE and Ofsted 
Guidance from exam boards 
Don’t know 


Other (please specify) 


Be ee ee ee 
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CPD 


3c. Please indicate, from the list below, what the training/CPD was based on. (Please tick all 
that apply) 


Exam board information 
Academic research 
Ideas from my school (e.g. internal INSET) 


Ideas from other schools 
Expertise of an external consultant 
Expertise of a programme provider 


Local authority/academy chain guidance 


Li) iif _ 


Other (please specify) 


Influence 


4. Please rate the level of influence that each of the following factors had on the decision to 
adopt your approach. (Please tick one box in each row) 


Strong Some No Not 
His tieugntneanpiaeehs jnflence influence influence applicable 
4 2 3 4 


...would be straightforward to implement 


...was likely to be popular with staff 


...was likely to be popular with parents 


...was likely to be popular with pupils 


... Was inexpensive 


Lit Li ia | 
Li Li Li 
Lit Lita i 
Li Li i | 


...was backed by academic research 
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...was a good fit with existing practices [| [| [| [| 
...aligned with our professional experience [ ] [ ] [| [| 


BLOCK C - Your general approach to teaching and learning to support pupils’ 
progress 


We would now like you to think more broadly about how you develop your teaching to support pupils 
progress. 


ConsultPP 


To what extent do you consult the following sources when deciding on your approaches 
to support pupils’ progress? (Please tick one box in each row) 


A lot A little Notat all 
1 2 3 


Pupil performance data 


External organisations (e.g. local authority, academy chain, DfE or 
Ofsted) 


Articles, reports, books or summaries based on academic research 
(paper or web based) 


Articles, reports, books or summaries based on teacher experience 
(paper or web based) 


Information gathered through training/CPD 


Online evidence platforms or databases (e.g. the Sutton Trust Teaching 
and Learning Toolkit) 


Guidance from exam boards 


Colleagues within my own school 


| ia LI Lie LL) ee 
1 Oe Lee CL) ee 
[1 ee Lie LL) ee 


Colleagues in other schools 
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UnderstandPP 


6. How easy do you find it to understand the information that these sources provide about how 
to support pupils’ progress? (Please tick one box in each row) 


Very Quite Notvery  Notatall | don’t use 
easy easy easy easy this source 
1 2 3 4 5 


Pupil performance data 


External organisations (e.g. local authority, 
academy chain, DfE or Ofsted) 


Articles, reports, books or summaries based 
on academic research (paper or web based) 


Articles, reports, books or summaries based 
on teacher experience (paper or web based) 


Information gathered through training/CPD 


Online evidence platforms or databases (e.g. 
the Sutton Trust Teaching and Learning 
Toolkit) 


Guidance from exam boards 


Colleagues within my own school 


1 OR aoe Lee 
1 Ua eee CL) ee 
1 UO ae CL) ee 
1 UR ae CL) ee 
i a ae CS) ee 


Colleagues in other schools 
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BLOCK D Evidence-based teaching and using evidence from research 
Evidence3 


7. What does the term ‘evidence-based teaching’ mean to you? 


Please select up to three boxes that best describe your understanding of the term. 


Conducting action research and applying the learning 
Learning from colleagues and applying the learning 


Applying Ofsted or DfE guidance 


Using an online evidence platform/database (e.g. Sutton Trust Toolkit) and applying 
the learning 


Applying exam board guidance 
Combining academic research evidence with my professional expertise 
Using pupil performance data to track pupil progress and plan ahead 


Applying the recommendations of an external supplier 


Reading and applying information from academic research or from working with 
researchers 


Learning from external consultants, trainers or advisors 


| don’t know 


i LO aL ee Lee Le 
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Research 


This question aims to find out how (if at all) you use research information in your work. By 
‘research’ we mean information from books, reports, articles, summaries, training or events that 
is based on academic studies. 


Please indicate the extent to which you agree or disagree with the following statements. (Please 
tick one box in each row). 


Strongly Agree Neither Disagree Strongly 
agree agree nor disagree 
disagree 
2 3 4 
1 5 


Information from research plays an important 
role in informing my/our teaching practice 


| do not believe that using information from 
research will help to improve pupil outcomes 


| know where to find relevant research that 
may help to inform teaching methods/practice 


My school leaders/governors do not 
encourage me to use information from 
research to improve my practice 


| am able to relate information from research to 
my context 


Other staff in my school rarely use information 
from research to inform their teaching practice 


| feel confident about analysing information 
from research 


Information from research conducted 
elsewhere is of limited value to our school 


| use information from research to help me to 
decide how to implement new approaches in 
the classroom 


[| Se 0 eee LL) ee LL ee 
[| See Lee Lee LL ee 
[| a Lee Ll) ee LL eee 
[| Se 1) ee Lee LL ee 
[| Bae |) ee Ll) ee LL eee 
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ResUsed 
9. In the last year, how (if at all) have you used information from academic research to inform 
your practice? (Please tick all that apply) 
Respondents can either tick ‘| have not used information from research...’ OR as 
many responses as they like from items 2 to 7. 


Respondents are routed as follows: 


‘| have not used information from research’ (item 1) — go to Q11. 


EITHER or BOTH ‘change classroom practice’ (item 4) /influenced 
colleagues to change their practice (item 6), regardless of whether they 
have ticked any other options — go to Q10 

Any combination of the remaining items BUT NOT INCLUDING EITHER OR 
BOTH OF items 4 and 6 —- go to Q11. 


L 


| have not used information from academic research in the last year 
Or, in the last year | have used information from academic research to: 


discuss best practice with colleagues 


reflect on my own practice 


change classroom practice (this could be starting, developing or discontinuing an 
approach) 


contribute to my own research/enquiry 


influence colleagues to change their classroom practice (this could be starting, 
developing or discontinuing an approach) 


mL) | 


improve my knowledge of a topic or subject 


{Select if ResUse = 4 and/or 6} 
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Change 


10. What was it about the research information that enabled you to change classroom 
practice? (Please tick all that apply) 


Question filtered from Q9 for respondents answering either or both of ‘changed 
classroom practice’ (item 4) / ‘influenced colleagues to change classroom practice 
(item 6). 


Multiple response item: respondents can select as many answers as they wish (no 
minimum). Online survey: statements to be randomised where technology allows, 
‘other’ should always appear last. 


It was clear (e.g. language, style, presentation) 


It was convincing 


| was able to discuss the research with a researcher or someone else who 
understood it 


| could see clearly how the research related to our context 


There was coaching and training available based on the research 


It contained practical guidance about how to apply the research in the classroom 


| was able to see the research being applied in another school 


It encouraged collaborative enquiry 


It was supported by resources (e.g. funding, materials) 


Lim ia | aL ee Ls 


Other (please specify) 


BLOCK E - Your knowledge about research 


In this section we would like to gather some information about your knowledge of research. Please 
answer the questions without referring to other sources. 


Knowledge 


11. Current understanding from academic research suggests that each of the following 
statements is ‘true’ or ‘false’. (Please tick the answer that you know to be correct in each row. 
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If you are not sure, please tick ‘don’t know’). 


True False Don’t know 

The research says that: 
1 2 3 

Drinking six to eight glasses of water per day improves pupil 
learning outcomes 


Reducing class size is one of the most cost-effective ways to 
improve pupil learning outcomes 


Extending the school day is more likely to improve learning 
outcomes for pupils on Free School Meals than pupils not on 
Free School Meals 


Interventions that focus solely on raising pupil aspirations have 
little impact on learning outcomes 


Setting pupils by ability improves learning outcomes for all 
pupils 


Individual pupils learn best when they receive information in 
their preferred learning style (e.g. auditory, visual, kinaesthetic) 


Peer tutoring (Students supporting other students with their 
learning) usually benefits the pupil being tutored more than the 
pupil doing the tutoring 


Homework has a greater impact on pupils’ learning outcomes 
at secondary school than at primary school 


UI Pa Um !ULhLhe || lh 
Be Oe | ee 
IT Um | ae | a 


Methods 


12. Below are descriptions of three reasons why someone would want to carry out research. 
Along the top of the table are five different research methods. 


Please match the research purpose with the best research method for achieving it by selecting the 
relevant option. Please select one box in each row. There are only three matches — two methods are 
incorrect (please do not use the same answer more than once). 


Randomised Longitudinal Interviews Literature Correlational 
Controlled study and/or review study 
Trial questionnaires 
2 4 5 
1 3 


To provide an overview of the 


evidence base C1 C C ia Cl 
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To determine whether an 
intervention or approach has a 


direct impact on pupil learning [J [J el i L 


outcomes 


To understand how an intervention 


or approach works in practice L] LJ L] L] LJ 
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BLOCK F - About the intervention 


The next questions relate to your schools participation in the Rochdale IPLCN: Research into 
Practice Project. 


You/your school has been invited to take part in the Rochdale IPLCN: Research into Practice 
Project. The following set of questions asks about your views on this initiative. 


Participation 


13. Did your school take part in the Rochdale IPLCN: Research into Practice Project? (Please 
tick one box only) 


Yes, and | had first-hand involvement [| 


Yes, my colleague/s were involved, and they shared the a 
learning with me 


Yes, my colleague/s were involved, but | don’t know any [J 
more about it 


No, my school did not take part [| 


I’m not sure [J 


The following questions ask you to comment on your involvement in the Rochdale IPLCN: Research 
into Practice Project. In these questions we would like you to think about your experiences of the 
whole project rather than its specific components. 


{Select if Participation = 1&2} 
Information1 


14. How did you feel about the amount of information that was provided through the project? 
(Please tick one box only) 


There was too much information [] 
The amount of information was about right |_| 


There was too little information [] 


{Select if Participation = 1&2} 
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Information2 


15. To what extent did the information from the project enable you to... (Please tick one box in 
each row) 


A lot A little Not at all 
discuss best practice with colleagues in my school L| LI L| 
share the learning with people or organisations outside my L | L| L| 


school 


reflect on my own practice L L L] 


change classroom practice (this could be starting, 
developing or discontinuing an approach) LI L LI 


reinforce existing practices [| L] L] 


conduct my own research or enquiry [| [ [| 


influence colleagues in my school to change their 
classroom practice (this could be starting, developing or LJ [| L | 
discontinuing an approach) 


improve my knowledge of a topic or subject [| iL] [ ] 


{Select if Participation = 1&2} 
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ResultQ 


16. To what extent do you agree or disagree with the following statements? (Please tick one box 
in each row) 


As a result of the project | have: Strongly Agree Neither Disagree Strongly 
agree agree nor disagree 
disagree 


Sought out further evidence, 
research or information about a al L] LJ LJ L] 


topic relevant to my practice. 


Sought out further evidence, 
research or information on other L] L] L] L] L] 


topics/approaches. 


In the following question we would like you to think about the individual elements of the Rochdale 
IPLCN: Research into Practice Project rather than the project as a whole. 


{Select if Participation = 1&2} 
Elements 


17. How would you rate the following elements of the Rochdale IPLCN: Research into Practice 
Project? (please tick one box in each row) 


Very Quite Average Quite Very Did not use/did 
good good poor poor not attend 


The Research _ into 

Practice events with 

presentations and CO = | i OJ L| 
interactive sessions 


The Research _ into 


Practice lead visiting [| [| [ ] [] L] L] 


your school 


Collaborative 


planning and | | LJ] [| L | L] 


feedback 


Individual 
support/advice from 


the Research _ into [| [| [ ] [] L] L] 


Practice lead via 
phone/email 
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{Select if Participation = 1&2} 
Recommend 


18a. Overall, would you recommend the Rochdale IPLCN: Research into Practice Project to 
another school? (Please tick one box only) 


Yes, definitely [] 


Yes, probably 


Definitely not 


CO 
Probably not CJ 
Oo 


{Select if Participation = 1&2} 
Reason 


18b. Please explain your reasons: 


Thank you very much for completing the survey. 
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Appendix D: Outcome measures 


The five outcomes measures used in the analysis were developed by Poet at al. (unpublished). 


Positive disposition to academic research in informing 
teaching practice 


Composite scale measures constructed using components of different survey questions were used to 
compare before and after measures. The first of these measures combined responses to six items: 


1. Ease of understanding reports, books, or summaries based on academic research (see 
Appendix C, Q6, item ay 

2. Information from research plays an important role in informing my/ our teaching practice (Q8, 
item 1); 

3. | know where to find relevant research that may help to inform teaching methods/ practice (Q8, 
item 3); 

4. |am able to relate information from research to my context (Q8, item 5); 

| feel confident about analysing information from research (Q8, item 7); 

6. | use information from research to help me decide how to implement new approaches in the 
classroom (Q8, item 9). 


on 


The Cronbach's alpha for outcome measure one was 0.80 at baseline and 0.85 at follow-up, indicating 
that the questions were highly correlated. 


Use of academic research to inform selection of teaching 
approaches 


The second composite measure was constructed by combining four questions: 


1. Articles, reports, books or summaries based on academic research (paper or web based) 
were important when identifying the approach (Q3b, item 4); 

2. Online evidence platforms or databases (e.g. the Sutton Trust Teaching and Learning Toolkit) 
were important when identifying the approach (Q3b, item 9); 

3. Influence the fact approach was backed by academic research had on decision to adopt it 
(Q4, item 6); 

4. The extent to which articles, reports, books or summaries based on academic research (paper 
or web based) are consulted when deciding on approaches to support pupil’s progress (Q5, 
item 3). 


This deviates from the NFER guidelines, which suggest the outcome measure is constructed with the 
questions above and, ‘CPD based on academic research was important when identifying approach in 
Q3a’, which is only included in the outcomes survey. By excluding this particular question it was 


7 Teachers who responded ‘I don’t use this source’ to question 6, item 3, have been assigned a score of 2.5 on 
this measure, as have those with missing values. This coding follows the evaluation guidelines set out in Poet et 
al. (unpublished). 
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possible to compare results across the two time points. The Cronbach's alpha for the measure was 
0.49 at baseline and 0.68 in the outcomes survey, therefore weakly correlated at baseline and 
moderately at follow-up. 


Perception that academic research is not useful to teaching 


Measure three captured teachers’ ‘perception that academic research is not useful to teaching’ 
combining responses to two survey questions: 


1. 1 do not believe that using information from research will help to improve pupil outcomes (Q8, 
item 2); 
2. Information from research conducted elsewhere is of limited value of our school (Q8, item 8). 


The Cronbach’s alpha for this measure was 0.55 at baseline and 0.65 in the outcomes survey, 
indicating a low to moderate correlation between indicators. 


Perception that own school does not encourage use of 
academic research 


The fourth outcome measure aimed to capture the ‘perception that the teachers’ own school does not 
encourage use of academic research’, combining responses to two items on question 8: 


1. My school leaders/ governors do not encourage me to use information from research to 
improve my practice (item 4); 

2. Other staff in my school rarely use information from research to inform their teaching practice 
(item 6). 


The Cronbach’s alpha for this outcome measure was lower than others, 0.34 at baseline and 0.35 in 
the outcomes survey, suggesting a weak correlation between measures, and lower reliability. Indeed 
the measure was not considered robust enough to be included in the analysis, and was therefore 
excluded from the report. 


Active engagement with online evidence platforms 


The measure constructed to explore ‘active engagement with online evidence platforms’ combined 
questions exploring the extent to which teachers used online evidence platforms, such as the Sutton 
Trust Teaching and Learning Toolkit (Q5, item 6) and how easily they understood them (Q6, item 6).*® 
The Cronbach’s alpha for the measure was 0.60 at baseline and 0.56 in the outcomes survey, 
showing a low to moderate correlation between the two measures. 


*8 Teachers who responded ‘I don’t use this source’ to question 6, item 6, have been assigned a score of 2.5 on 
this measure, as have those with missing data. This coding follows the evaluation guidelines set out in Poet et al. 
(unpublished). 
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